Software testing can be an important step during the software development cycle, e.g., that helps verify whether the software meets defined requirements and works as expected. In modern software development techniques such as the iterative and incremental agile software development approaches, software testing is a continuing process. Compared to traditional software development processes, in which most of the testing efforts oftentimes occur after the software coding stage has been completed, in iterative and agile techniques, the software is tested continuously before it is released. The agile approach generally involves software programmers testing the software completely after each software enhancement or change, e.g., to help ensure the software works as intended and is deliverable to a customer and/or to a production environment after each development cycle. Such development approaches therefore can require frequent execution of automated testing. For example, with the iterative and/or agile approaches, it may be necessary for a programmer to test the software each time a feature is coded or completed, every time code is checked into a version control system, etc.
Test automation or in general software testing methods can be distinguished into white- and black-box tests. White-box tests use knowledge about the internal structure and workings of the software and can check if internal application programming interfaces (APIs) work as expected or whether the internal state of the program is correct. White-box tests can be applied at the unit, integration, and system levels of the software testing process.
Black-box tests require no knowledge about the internal implementation and treat the software under test like a “black box”. Black-box tests may be performed in addition to white-box tests, and may also be applied to the acceptance level of software testing. One kind of acceptance tests is the graphical user interface (GUI) test. A GUI test involves program code that tries to reproduce the interaction that is performed by the user on the interface, e.g., by clicking with the mouse on controls, by entering any kind of input via the keyboard, etc. After the interaction part is finished, the test code compares the output results with the expected results. This code can be written manually by the programmer or automatically recorded by using corresponding automation tools that allow recording of all interactions on the interface and transforming them into a corresponding code (e.g., playback script).
GUI tests have the disadvantage that they are very sensitive to changes on the user interface (UI). For example, GUI tests would require frequent changes to the test code because the agile approach demands very frequent changes (iterations) of the software. As a solution to such problems, it is useful to implement GUI tests that are realized as black-box as well as white-box tests. The software pattern known as the “model view presenter,” which is a common approach for the implementation of modern web applications, follows this idea.
Most of the existing frameworks for web browser graphical user interface tests (e.g., Selenium) are divided into a component that offers an application programming interface (API) to define the interaction and to check the expected result, as well as a component that executes the defined interaction in the web browser. The component that executes the defined test interactions use the native automation APIs provided by the browsers. These native APIs of the browser allow checking the state of the document object model (DOM) structure in the browser. As such, this approach enables to check the internal state of an application and to implement white-box tests.
Modern web applications, which are frequently implemented in JavaScript, are optimized by a pre-compiler step. Some or all of the following and/or other common pre-compiler steps can make it difficult and sometimes even impossible for existing test frameworks to test the production system: code translation/generation (e.g., from Java to JavaScript), creation of code variations (e.g., for cross-browser support), code optimization, obfuscation of code, etc.
Such changes to the code during compiling and/or obfuscation may cause the behavior of the application to dramatically change in a production environment. In addition, all these steps result in an auto-generated production code that differs from the original code. Because of the code translation, generation, and the obfuscating steps, the code structure itself may be changed, e.g., possibly involving changes to function and variable names, etc. Thus, testing such compiled and/or obfuscated application code on production environments using existing white-box test frameworks can become very difficult and sometimes even impossible, not to mention expensive.
It thus will be appreciated that existing approaches to white-box testing of GUI web applications are disadvantageous to the extent that they require additional tools (software) to test the installed production environment, with such test tools often being platform- and/or target-system specific; quite often require that test code be written in a different programming language as the implementation language of the production code; are difficult for the end user to execute the test cases to validate his/her environment (e.g., after an installation to verify if the environment works correctly); etc.
Certain example embodiments help address these and/or other issues. For instance, an example method embodiment for testing an application includes, responsive to a first parameter received from a test suite executer, executing test bootstrap code included in the application, where the application and the test suite executer are running in a browser. The method also includes the executed test bootstrap code retrieving a test case to be run on the application and a test case executer to execute the test case, executing the retrieved test case in a same context of the browser as the application, and storing results from executing the retrieved test case.
According to an aspect of certain example embodiments, the test bootstrap code is compiled together with code of the application.
According to another aspect of certain example embodiments, the retrieving includes requesting, by the executed test bootstrap code, the test case from a remote computer; responsive to the requesting, receiving the test case and the test case executer; and injecting the received test case and the test case executer to the application.
In another aspect of certain example embodiments, executing the retrieved test case includes performing the execution of the test case by the injected test case executer.
The method may include calling one or more internal functions of the application from within the injected test case code, in certain example embodiments.
The method may include starting the test suite executer in the browser, where the starting includes loading code of the test suite executer from a computer, injecting the loaded code of the test suite executer to a page of the browser, and executing the injected code of the test suite executer to provide the first parameter to the application.
An example computer system embodiment for testing an application includes first processing resources including at least one processor and a memory. The processor is configured to perform, responsive to a first parameter received from a test suite executer, executing test bootstrap code included in the application, where the application and the test suite executer are running in a browser. The processor is further configured to perform the executed test bootstrap code retrieving a test case to be run on the application and a test case executer to execute the test case, executing the retrieved test case in a same context of the browser as the application, and storing results from executing the retrieved test case.
Another example computer system embodiment includes second processing resources communicatively connected to the first processing resources over a network, and storage connected to the second processing resources. The storage stores a plurality of test cases, each said test case including (a) interactions between the application and components external to the application that were gathered during a run of the application and received user input from a user, and (b) assertions to be made to the application once the application reaches a desired test state. Retrieving the test case includes requesting, by the executed test bootstrap code, the test case from the second processing resources, responsive to the requesting, receiving the test case and the test case executer from the storage, and injecting the received test case and the test case executer to the application.
An example non-transitory computer readable storage medium includes instructions stored thereon, which when executed by a computer, causes the computer to perform operations for testing an application. The operations include, responsive to a first parameter received from a test suite executer, execute test bootstrap code included in the application, where the application and the test suite executer are running in a browser. The operations also include the executed test bootstrap code retrieving a test case to be run on the application and a test case executer to execute the test case, executing the retrieved test case in a same context of the browser as the application, and storing results from executing the retrieved test case.
These aspects, features, and example embodiments may be used separately and/or applied in various combinations to achieve yet further embodiments.